roachprod: bump local Prometheus version to 2.53.5#170676
Merged
Merged
Conversation
The local Prometheus version (2.27.1) is over 4 years old and has been observed crashing in CI (e.g. cockroachdb#169503 on s390x). This commit bumps it to 2.53.5, the latest patch in the 2.53 LTS series. We choose to stay on 2.x instead of upgrading to 3.x to avoid any breaking changes. The 2.x bump is sufficient to address the stability issue mentioned above and will act as an drop-in replacement before we move away from Prometheus entirely (VictoriaMetrics). The 2.53.5 tarballs have been uploaded to gs://cockroach-test-artifacts/prometheus/ for amd64, arm64, and s390x. No testdata regeneration was required because PrometheusVersion is not embedded in any cloud-provider startup script. Resolves: cockroachdb#169559 See also: cockroachdb#169503 Release note: None
Contributor
|
😎 Merged successfully - details. |
Member
|
This change is |
kyleli03
marked this pull request as ready for review
May 21, 2026 13:55
kyleli03
requested review from
srosenberg and
williamchoe3
and removed request for
a team
May 21, 2026 13:55
golgeek
approved these changes
May 21, 2026
golgeek
left a comment
golgeek
left a comment
Contributor
There was a problem hiding this comment.
We'll be running a release that's only 11 months old! 🚀
It's probably worth it to backport this to all supported branches.
Contributor
|
I trusted you, but "trust, but verify". Just a nit, for a change like this I think posting some proof that the binary was updated would be nice for the reviewers just so we don't need to dig it up from the ci test artifacts. I also could do a better job at this so feel free to complain to me about this in my PRs too. test.log ➜ ~ gcloud storage ls --long 'gs://cockroach-test-artifacts/prometheus/prometheus-2.53.5.linux-*.tar.gz'
106271029 2026-05-20T18:09:17Z gs://cockroach-test-artifacts/prometheus/prometheus-2.53.5.linux-amd64.tar.gz
99932033 2026-05-20T18:09:24Z gs://cockroach-test-artifacts/prometheus/prometheus-2.53.5.linux-arm64.tar.gz
104923573 2026-05-20T18:09:31Z gs://cockroach-test-artifacts/prometheus/prometheus-2.53.5.linux-s390x.tar.gz |
williamchoe3
approved these changes
May 21, 2026
williamchoe3
added
the
backport-all
Contributor
Author
According to activity it looks like 26.2, 26.1, 25.4, 25.2, 24.3, and 24.1 are supported. On the release support policy however it shows that 24.3 and 24.1 have passed their date for maintenance support. Would 24.3 and 24.1 be considered valid for backport then in this case? |
Contributor
|
@Dev-Kyle I added the backport-all tag to this PR, once this merges, PRs will attempt to be created that target all previously supported releases. For a change like this, I think it should work without any issues, but for larger changes, it may fail and in that case you'd need to backport manually. |
Contributor
Author
|
TFTR! /trunk merge |
Contributor
|
FYI for back porting versions, here's the guidance from the backport claude skill
"Only backport to versions whose Maintenance Support end date has not yet https://www.cockroachlabs.com/docs/releases/release-support-policy#supported-versions so no 24.1 according to that policy |
This was referenced May 21, 2026
|
Encountered an error creating backports. Some common things that can go wrong:
You might need to create your backport manually using the backport tool. 💡 Consider backporting to the fork repo instead of the main repo. See instructions for more details. 💡 Consider backporting to the fork repo instead of the main repo. See instructions for more details. 💡 Consider backporting to the fork repo instead of the main repo. See instructions for more details. 💡 Consider backporting to the fork repo instead of the main repo. See instructions for more details. Branch release-26.3 doesn't exist, skipping Backport to branch release-26.3 failed. See errors above. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
trunk-io Bot
pushed a commit
that referenced
this pull request
May 27, 2026
CollectPoint and CollectInterval treated any non-empty Prometheus warnings slice as a fatal error. With Prometheus 2.53+, this slice now includes informational annotations like PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "sys_host_disk_write_bytes" which is emitted for queries such as rate(sys_host_disk_write_bytes[1m]) and is purely a naming-convention nudge -- the query result is still valid. After the recent bump to Prometheus 2.53.5 (#170676), this broke admission-control/disk-bandwidth-limiter outright (it t.Fatals on collection errors) and dropped bandwidth samples in admission- control/{index,single-node-index}-backfill, which already log the error and continue. Extract a handlePromWarnings helper that classifies entries by the "PromQL info:" wire-format prefix: - "PromQL info: ..." entries are logged and dropped. - "PromQL warning: ..." entries, and any unprefixed entries (e.g. legacy remote-read warnings, which predate the PromQL annotation system), continue to surface as the same error string as before. Prometheus' annotations package defines exactly two sentinel error types (PromQLInfo and PromQLWarning), so the prefix check is the actual wire-format contract; the client_golang v1 API we use flattens both into a single []string with no structured alternative. Resolves: #170841 See also: #170790, #170793, #170843 Release note: None
blathers-crl Bot
pushed a commit
that referenced
this pull request
May 28, 2026
CollectPoint and CollectInterval treated any non-empty Prometheus warnings slice as a fatal error. With Prometheus 2.53+, this slice now includes informational annotations like PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "sys_host_disk_write_bytes" which is emitted for queries such as rate(sys_host_disk_write_bytes[1m]) and is purely a naming-convention nudge -- the query result is still valid. After the recent bump to Prometheus 2.53.5 (#170676), this broke admission-control/disk-bandwidth-limiter outright (it t.Fatals on collection errors) and dropped bandwidth samples in admission- control/{index,single-node-index}-backfill, which already log the error and continue. Extract a handlePromWarnings helper that classifies entries by the "PromQL info:" wire-format prefix: - "PromQL info: ..." entries are logged and dropped. - "PromQL warning: ..." entries, and any unprefixed entries (e.g. legacy remote-read warnings, which predate the PromQL annotation system), continue to surface as the same error string as before. Prometheus' annotations package defines exactly two sentinel error types (PromQLInfo and PromQLWarning), so the prefix check is the actual wire-format contract; the client_golang v1 API we use flattens both into a single []string with no structured alternative. Resolves: #170841 See also: #170790, #170793, #170843 Release note: None
blathers-crl Bot
pushed a commit
that referenced
this pull request
May 28, 2026
CollectPoint and CollectInterval treated any non-empty Prometheus warnings slice as a fatal error. With Prometheus 2.53+, this slice now includes informational annotations like PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "sys_host_disk_write_bytes" which is emitted for queries such as rate(sys_host_disk_write_bytes[1m]) and is purely a naming-convention nudge -- the query result is still valid. After the recent bump to Prometheus 2.53.5 (#170676), this broke admission-control/disk-bandwidth-limiter outright (it t.Fatals on collection errors) and dropped bandwidth samples in admission- control/{index,single-node-index}-backfill, which already log the error and continue. Extract a handlePromWarnings helper that classifies entries by the "PromQL info:" wire-format prefix: - "PromQL info: ..." entries are logged and dropped. - "PromQL warning: ..." entries, and any unprefixed entries (e.g. legacy remote-read warnings, which predate the PromQL annotation system), continue to surface as the same error string as before. Prometheus' annotations package defines exactly two sentinel error types (PromQLInfo and PromQLWarning), so the prefix check is the actual wire-format contract; the client_golang v1 API we use flattens both into a single []string with no structured alternative. Resolves: #170841 See also: #170790, #170793, #170843 Release note: None
blathers-crl Bot
pushed a commit
that referenced
this pull request
May 28, 2026
CollectPoint and CollectInterval treated any non-empty Prometheus warnings slice as a fatal error. With Prometheus 2.53+, this slice now includes informational annotations like PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "sys_host_disk_write_bytes" which is emitted for queries such as rate(sys_host_disk_write_bytes[1m]) and is purely a naming-convention nudge -- the query result is still valid. After the recent bump to Prometheus 2.53.5 (#170676), this broke admission-control/disk-bandwidth-limiter outright (it t.Fatals on collection errors) and dropped bandwidth samples in admission- control/{index,single-node-index}-backfill, which already log the error and continue. Extract a handlePromWarnings helper that classifies entries by the "PromQL info:" wire-format prefix: - "PromQL info: ..." entries are logged and dropped. - "PromQL warning: ..." entries, and any unprefixed entries (e.g. legacy remote-read warnings, which predate the PromQL annotation system), continue to surface as the same error string as before. Prometheus' annotations package defines exactly two sentinel error types (PromQLInfo and PromQLWarning), so the prefix check is the actual wire-format contract; the client_golang v1 API we use flattens both into a single []string with no structured alternative. Resolves: #170841 See also: #170790, #170793, #170843 Release note: None
blathers-crl Bot
pushed a commit
that referenced
this pull request
May 28, 2026
CollectPoint and CollectInterval treated any non-empty Prometheus warnings slice as a fatal error. With Prometheus 2.53+, this slice now includes informational annotations like PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "sys_host_disk_write_bytes" which is emitted for queries such as rate(sys_host_disk_write_bytes[1m]) and is purely a naming-convention nudge -- the query result is still valid. After the recent bump to Prometheus 2.53.5 (#170676), this broke admission-control/disk-bandwidth-limiter outright (it t.Fatals on collection errors) and dropped bandwidth samples in admission- control/{index,single-node-index}-backfill, which already log the error and continue. Extract a handlePromWarnings helper that classifies entries by the "PromQL info:" wire-format prefix: - "PromQL info: ..." entries are logged and dropped. - "PromQL warning: ..." entries, and any unprefixed entries (e.g. legacy remote-read warnings, which predate the PromQL annotation system), continue to surface as the same error string as before. Prometheus' annotations package defines exactly two sentinel error types (PromQLInfo and PromQLWarning), so the prefix check is the actual wire-format contract; the client_golang v1 API we use flattens both into a single []string with no structured alternative. Resolves: #170841 See also: #170790, #170793, #170843 Release note: None
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The local Prometheus version (2.27.1) is over 4 years old and has been observed crashing in CI (e.g. #169503 on s390x). This commit bumps it to 2.53.5, the latest patch in the 2.53 LTS series.
We choose to stay on 2.x instead of upgrading to 3.x to avoid any breaking changes. The 2.x bump is sufficient to address the stability issue mentioned above and will act as an drop-in replacement before we move away from Prometheus entirely (VictoriaMetrics).
The 2.53.5 tarballs have been uploaded to
gs://cockroach-test-artifacts/prometheus/ for amd64, arm64, and s390x. No testdata regeneration was required because PrometheusVersion is not embedded in any cloud-provider startup script.
Validation
Manually triggered
admission-control/intent-resolutionagainst this branch via TeamCity:https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel?branch=170676&buildTypeTab=overview&mode=builds
Resolves: #169559
See also: #169503
Release note: None